Configurable and incremental database migration framework for heterogeneous databases

ABSTRACT

Various methods are provided for facilitating heterogeneous database migration. One example method may comprise accessing a migration configuration file, the migration configuration file comprising information indicative of at least a source database type of a source database, connection information of accessing the source database, a destination database type of a destination database, and connection information for accessing the destination database, generating configuration data based on the source database type and the destination database type to generate a mapping of a source database table stored within the source database to a destination database table stored within the destination database, and generating a query for execution on the source database enabling data migration from the source database to the destination database.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit under 35 USC 119 (e) of U.S. provisionalApplication No. 62/315,086, filed Mar. 30, 2016, the entire content ofwhich is incorporated herein by reference.

FIELD

Embodiments of the invention relate, generally, to facilitatingheterogeneous database migration, and more specifically for a system,method, apparatus, and computer program product for providing aconfigurable and incremental database migration framework forheterogeneous databases.

BACKGROUND

The applicant has discovered problems with current methods, systems, andapparatuses for providing heterogeneous database migration. Throughapplied effort, ingenuity, and innovation, Applicant has solved many ofthese identified problems by developing a solution that is embodied bythe present invention, which is described in detail below.

BRIEF SUMMARY

In general, embodiments of the present invention provided herein includesystems, methods and computer readable media for facilitating improvedheterogeneous database migration. Some embodiments provide for morespecifically to a system, method, apparatus, and computer programproduct for providing a configurable and incremental database migrationframework to facilitate heterogeneous database migration.

In some embodiments, an apparatus for facilitating heterogeneousdatabase migration may be provided, the apparatus comprising at leastone processor and at least one memory including computer program code,the at least one memory and the computer program code configured to,with the processor, cause the apparatus to at least access a migrationconfiguration file, the migration configuration file comprisinginformation indicative of at least a source database type of a sourcedatabase, connection information of accessing the source database, adestination database type of a destination database, and connectioninformation for accessing the destination database, generateconfiguration data based on the source database type and the destinationdatabase type to generate a mapping of a source database table storedwithin the source database to a destination database table stored withinthe destination database, and generate a query for execution on thesource database enabling data migration from the source database to thedestination database.

In some embodiments, the first database type is different than thesecond database type. In some embodiments, the at least one memory andthe computer program code are further configured to, with the processor,cause the apparatus to execute the query on the source database toextract data, frame the data into an insert format, and cause storage ofthe data, in the insert format, in the destination database. In someembodiments, the at least one memory and the computer program codeconfigured for framing the query is further configured to, with theprocessor, cause the apparatus to call a function for manipulating datavalues stored in the source database table or any data exchange formatsin advance of causing storage of the data in the destination databasetable. In some embodiments, the at least one memory and the computerprogram code configured for framing the query is further configured to,with the processor, cause the apparatus to call a function configuredfor supporting data migration from a plurality of source databasetables, the function configured for supporting data migration from theplurality of source database tables comprising receiving informationindicative of each of the plurality of source database table, receivinginformation indicative of the destination database, and an order inwhich the plurality of source database tables join in the destinationdatabase.

In some embodiments, the at least one memory and the computer programcode are further configured to, with the processor, cause the apparatusto populate temporary tables in an instance in which a table is havingprimary key to maintain referential integration.

In some embodiments, the at least one memory and the computer programcode are further configured to, with the processor, cause the apparatusto preceding accessing the migration configuration file, generate themigration configuration file, wherein generation of the

migration configuration file comprises receiving source databaseconnection information, receiving destination database connectioninformation, and confirming connectivity to the source database and thedestination database.

In some embodiments, a method for facilitating heterogeneous databasemigration may be provided, the method comprising accessing a migrationconfiguration file, the migration configuration file comprisinginformation indicative of at least a source database type of a sourcedatabase, connection information of accessing the source database, adestination database type of a destination database, and connectioninformation for accessing the destination database, generatingconfiguration data based on the source database type and the destinationdatabase type to generate a mapping of a source database table storedwithin the source database to a destination database table stored withinthe destination database, and generating a query for execution on thesource database enabling data migration from the source database to thedestination database.

In some embodiments, the first database type is different than thesecond database type. In some embodiments, the method may furthercomprise executing the query on the source database to extract data,framing the data into an insert format, and causing storage of the data,in the insert format, in the destination database. In some embodiments,framing the query comprises calling a function for manipulating datavalues stored in the source database table or any data exchange formatsin advance of causing storage of the data in the destination databasetable. In some embodiments, framing the query comprises calling afunction configured for supporting data migration from a plurality ofsource database tables, the function configured for supporting datamigration from the plurality of source database tables comprisingreceiving information indicative of each of the plurality of sourcedatabase table, receiving information indicative of the destinationdatabase, and an order in which the plurality of source database tablesjoin in the destination database.

In some embodiments, the method may further comprise populatingtemporary tables in an instance in which a table is having primary keyto maintain referential integration. In some embodiments, the method mayfurther comprise preceding accessing the migration configuration file,generating the migration configuration file, wherein generation of themigration configuration file comprises receiving source databaseconnection information, receiving destination database connectioninformation, and confirming connectivity to the source database and thedestination database.

In some embodiments, a computer program product for facilitatingheterogeneous database migration may be provided, the computer programproduct comprising at least one non-transitory computer-readable storagemedium having computer-executable program code instructions storedtherein, the computer-executable program code instructions comprisingprogram code instructions for accessing a migration configuration file,the migration configuration file comprising information indicative of atleast a source database type of a source database, connectioninformation of accessing the source database, a destination databasetype of a destination database, and connection information for accessingthe destination database, generating configuration data based on thesource database type and the destination database type to generate amapping of a source database table stored within the source database toa destination database table stored within the destination database, andgenerating a query for execution on the source database enabling datamigration from the source database to the destination database.

In some embodiments, the first database type is different than thesecond database type. In some embodiments, the computer-executableprogram code instructions further comprise program code instructions forexecuting the query on the source database to extract data, framing thedata into an insert format, and causing storage of the data, in theinsert format, in the destination database. In some embodiments, thecomputer-executable program code instructions configured for framing thequery further comprise program code instructions for calling a functionfor manipulating data values stored in the source database table or anydata exchange formats in advance of causing storage of the data in thedestination database table. In some embodiments, the computer-executableprogram code instructions configured for framing the query furthercomprise program code instructions for calling a function configured forsupporting data migration from a plurality of source database tables,the function configured for supporting data migration from the pluralityof source database tables comprising receiving information indicative ofeach of the plurality of source database table, receiving informationindicative of the destination database, and an order in which theplurality of source database tables join in the destination database.

In some embodiments, the computer-executable program code instructionsfurther comprise program code instructions for populating temporarytables in an instance in which a table is having primary key to maintainreferential integration. In some embodiments, the computer-executableprogram code instructions further comprise program code instructions forpreceding accessing the migration configuration file, generating themigration configuration file, wherein generation of the migrationconfiguration file comprises receiving source database connectioninformation, receiving destination database connection information, andconfirming connectivity to the source database and the destinationdatabase.

BRIEF DESCRIPTION OF THE DRAWINGS

Having described certain example embodiments of the present disclosurein general terms above, reference will now be made to the accompanyingdrawings, which are not necessarily drawn to scale.

FIG. 1 shows an example system within which embodiments of the presentinvention may operate.

FIG. 2 shows a block diagram showing an example device for implementinga configurable and incremental database migration framework tofacilitate heterogeneous database migration using special-purposecircuitry in accordance with some exemplary embodiments of the presentinvention.

FIG. 3 shows a flow diagram depicting an exemplary embodiment ofperforming database migration, in accordance with some exemplaryembodiments of the present invention.

FIGS. 4 and 5 show flow diagrams depicting exemplary methods forimplementing a configurable and incremental database migration frameworkto facilitate heterogeneous database migration in accordance with someexemplary embodiments of the present invention.

DETAILED DESCRIPTION Overview

Various embodiments of the present invention are directed to improvedsystems, apparatuses, methods, and computer readable media forheterogeneous database migration. In this regard, embodiments of thepresent invention provide systems, devices, methods, and computerreadable media for providing a configurable and incremental databasemigration framework to facilitate heterogeneous database migration.

That is, by being provided connection information and a database type ofthe source and destination databases (MySQL, PostgreSQL, or the like),embodiments of the present invention may perform database migration.Whereas conventionally, a user would have to know, for examplesemantically, how to perform data manipulation in both the source anddestination database, features of the present invention include aframework which can be configurable and developed in any programminglanguage/independent of programming language. The present invention doesnot require a user to possess any database knowledge for execution.Furthermore, the present invention may be configured to callconfigurable functions which can have complex logic, whereas the samecannot be achieved through stored procedures. The present invention maybe further configured such that changes are easy to accommodate in that,for example, a user does not need to login to a server to make changesor execute. The present invention may further be configured such thatprimary key and foreign key constrains are maintained within anintermediate temporary table. Migration may be divided into subpartsand, in some embodiments, executed in many parallel threads based onlimit and offset.

Furthermore, the present invention may be configured such that, forexample, CSV, XML, or JSON files may be imported into database using thesame configurations. Moreover, the present invention may be configuredsuch that, based on the logs and/or status, when failure, a pointer canbe set to specific record from where the migration to start again.Furthermore, the present invention may be configured such that data maybe migrated to multiple and different target databases at the same time.

Definitions

Embodiments of the present invention now will be described more fullyhereinafter with reference to the accompanying drawings, in which some,but not all embodiments of the inventions are shown. Indeed, embodimentsof the invention may be embodied in many different forms and should notbe construed as limited to the embodiments set forth herein, rather,these embodiments are provided so that this disclosure will satisfyapplicable legal requirements. Like numbers refer to like elementsthroughout.

As used herein, the terms “data,” “content,” “information,” and similarterms may be used interchangeably to refer to data capable of beingtransmitted, received, and/or stored in accordance with embodiments ofthe present invention. Thus, use of any such terms should not be takento limit the spirit and scope of embodiments of the present invention.

As used herein, the term “database” may refer to any organizedcollection of data. A database, as referred to herein, is organized suchthat information may be accessed via a query (e.g., the location of eachof a plurality of consumers or merchants may be organized such that eachconsumer in a particular location may be queried).

As used herein, the term “database type” may refer to any databasemanagement system. For example, a database type may refer to a computersoftware application that interacts with the user, other applications,and the database itself to capture and analyze data (e.g., conventionaldatabase management software such as MySQL, POSTGRES or the like)

Technical Underpinnings and Implementation of Exemplary Embodiments

Though data migration has become an integral and necessary part of ITdepartment operations in today's business environment, it often causesmajor disruptions as a result of data quality and applicationperformance problems. Encountering these problems and subsequentlyidentifying solutions can severely impact budgets. Unfortunately datamigration options are limited and particularly so between heterogeneousdatabases. For example, due to the speed at which new technologies arecreated and introduced to the market, backwards compatibility to legacytechnologies is often not considered. That is, data created and storedby existing legacy technologies are burdened by not only their inheritlimitations, but also by lack of compatibility with existingtechnologies. However, as demand for data analysis and reportingcontinues to grow, the data created and stored by existing legacytechnologies is needed for completeness. For example, data related tohealth care systems and financial institutions will need be preserved asthe newer systems are created and implemented. Moreover, the time andmoney invested in maintaining and supporting legacy systems is notsustainable.

While some solutions do exist (e.g., software bridges used to facilitatetransition away from older database technologies), several limitationspreclude their practical viability. For instance, any software bridgerequires a thorough knowledge of both systems and, for each migration, acomplete work up is necessary based on the particulars. As such, thecost of such a solution is often prohibitory. Hence there exists a needfor a simpler, more intuitive system, which can be used with anycombination of heterogeneous databases, not require expert knowledge ofboth the legacy and new technology, and provide integration to newertechnology.

System Architecture

Methods, apparatuses, and computer program products of the presentinvention may be embodied by any of a variety of devices. For example,the method, apparatus, and computer program product of an exampleembodiment may be embodied by a networked device, such as a server orother network entity, configured to communicate with one or moredevices, such as one or more client devices. Additionally oralternatively, the computing device may include fixed computing devices,such as a personal computer or a computer workstation. Still further,example embodiments may be embodied by any of a variety of mobileterminals, such as a portable digital assistant (PDA), mobile telephone,smartphone, laptop computer, tablet computer, or any combination of theaforementioned devices.

In this regard, FIG. 1 discloses an example computing system withinwhich embodiments of the present invention may operate. Users may accessany of one or more first services 102A-N and/or any of one or moresecond services 110A-N via a network 116 (e.g., the Internet, or thelike) using a migration engine 108. Moreover, the first service 102A-Nmay comprise a server 104 in communication with a source database 106and the one or more second services 110A-N may comprise servers 112A-Nin communication with target databases 114A-N.

Both the server 104 and servers 112A-N may be embodied as a computer orcomputers as known in the art. The servers 104 and 112A-N may providefor receiving of electronic data from various sources, including but notnecessarily limited to the migration engine 108. Generally, the server104 may facilitate e-commerce transactions based on transactioninformation provided by various merchant devices or the like.

Databases 106 and 114A-N may be embodied as a data storage device suchas a Network Attached Storage (NAS) device or devices, or as a separatedatabase server or servers. Databases 106 and 114A-N may includeinformation accessed and stored by the servers 104 and 112A-N tofacilitate the operations of the services 102A-N and 110A-Nrespectively. For example, the databases 106 and 114A-N may include,without limitation, user account credentials for system administrators,merchants, and consumers, data indicating the products and promotionsoffered by the service, clickstream data, analytic results, reports,financial data, and/or the like.

Migration engine 108 may be any computing device as known in the art andoperated by a user. In embodiments where migration engine 108 is amobile device, such as a smart phone or tablet, the migration engine 108may execute an “app” to interact with any of services 102A-N. Such appsare typically designed to execute on mobile devices, such as tablets orsmartphones. For example, an app may be provided that executes on mobiledevice operating systems such as Apple Inc.'s iOS®, Google Inc.'sAndroid®, or Microsoft Inc.'s Windows 8®. These platforms typicallyprovide frameworks that allow apps to communicate with one another andwith particular hardware and software components of mobile devices. Forexample, the mobile operating systems named above each provideframeworks for interacting with location services circuitry, wired andwireless network interfaces, user contacts, and other applications in amanner that allows for improved interactions between apps while alsopreserving the privacy and security of consumers. In some embodiments, amobile operating system may also provide for improved communicationinterfaces for interacting with external devices (e.g., home automationsystems, indoor navigation systems, and the like). Communication withhardware and software modules executing outside of the app is typicallyprovided via application programming interfaces (APIs) provided by themobile device operating system.

Additionally or alternatively, migration engine 108 may interact throughthe any of services 102A-N via a web browser. As yet another example,migration engine 108 may include various hardware or firmware designedto interface with any of services 102A-N (e.g., where the migrationengine 108 is a purpose-built device offered for the primary purpose ofcommunicating with any of services 102A-N).

Example Apparatus for Implementing Embodiments of the Present Invention

Migration engine 108 may be embodied by one or more computing systems,such as apparatus 200 shown in FIG. 2. As illustrated in FIG. 2, theapparatus 200 may include a processor 202, a memory 204, input/outputcircuitry 206, communications circuitry 208, and migration module 210.The apparatus 200 may be configured to execute the operations describedabove with respect to FIG. 1 and below with respect to FIGS. 3 and 4.Although these components 202-210 are described with respect tofunctional limitations, it should be understood that the particularimplementations necessarily include the use of particular hardware. Itshould also be understood that certain of these components 202-210 mayinclude similar or common hardware. For example, two sets of circuitrymay both leverage use of the same processor, network interface, storagemedium, or the like to perform their associated functions, such thatduplicate hardware is not required for each set of circuitry. The use ofthe term “circuitry” as used herein with respect to components of theapparatus should therefore be understood to include particular hardwareconfigured to perform the functions associated with the particularcircuitry as described herein.

The term “circuitry” should be understood broadly to include hardwareand, in some embodiments, software for configuring the hardware. Forexample, in some embodiments, “circuitry” may include processingcircuitry, storage media, network interfaces, input/output devices, andthe like. In some embodiments, other elements of the apparatus 200 mayprovide or supplement the functionality of particular circuitry. Forexample, the processor 202 may provide processing functionality, thememory 204 may provide storage functionality, the communicationscircuitry 208 may provide network interface functionality, and the like.

In some embodiments, the processor 202 (and/or co-processor or any otherprocessing circuitry assisting or otherwise associated with theprocessor) may be in communication with the memory 204 via a bus forpassing information among components of the apparatus. The memory 204may be non-transitory and may include, for example, one or more volatileand/or non-volatile memories. In other words, for example, the memorymay be an electronic storage device (e.g., a computer readable storagemedium). The memory 204 may be configured to store information, data,content, applications, instructions, or the like, for enabling theapparatus to carry out various functions in accordance with exampleembodiments of the present invention.

The processor 202 may be embodied in a number of different ways and may,for example, include one or more processing devices configured toperform independently. Additionally or alternatively, the processor mayinclude one or more processors configured in tandem via a bus to enableindependent execution of instructions, pipelining, and/ormultithreading. The use of the term “processing circuitry” may beunderstood to include a single core processor, a multi-core processor,multiple processors internal to the apparatus, and/or remote or “cloud”processors.

In an example embodiment, the processor 202 may be configured to executeinstructions stored in the memory 204 or otherwise accessible to theprocessor. Alternatively or additionally, the processor may beconfigured to execute hard-coded functionality. As such, whetherconfigured by hardware or software methods, or by a combination thereof,the processor may represent an entity (e.g., physically embodied incircuitry) capable of performing operations according to an embodimentof the present invention while configured accordingly. Alternatively, asanother example, when the processor is embodied as an executor ofsoftware instructions, the instructions may specifically configure theprocessor to perform the algorithms and/or operations described hereinwhen the instructions are executed.

In some embodiments, the apparatus 200 may include input/outputcircuitry 206 that may, in turn, be in communication with processor 202to provide output to the user and, in some embodiments, to receive anindication of a user input. The input/output circuitry 206 may comprisea user interface and may include a display and may comprise a web userinterface, a mobile application, a client device, a kiosk, or the like.In some embodiments, the input/output circuitry 206 may also include akeyboard, a mouse, a joystick, a touch screen, touch areas, soft keys, amicrophone, a speaker, or other input/output mechanisms. The processorand/or user interface circuitry comprising the processor may beconfigured to control one or more functions of one or more userinterface elements through computer program instructions (e.g., softwareand/or firmware) stored on a memory accessible to the processor (e.g.,memory 204, and/or the like).

The communications circuitry 208 may be any means such as a device orcircuitry embodied in either hardware or a combination of hardware andsoftware that is configured to receive and/or transmit data from/to anetwork and/or any other device, circuitry, or module in communicationwith the apparatus 200. In this regard, the communications circuitry 208may include, for example, a network interface for enablingcommunications with a wired or wireless communication network. Forexample, the communications circuitry 208 may include one or morenetwork interface cards, antennae, buses, switches, routers, modems, andsupporting hardware and/or software, or any other device suitable forenabling communications via a network. Additionally or alternatively,the communication interface may include the circuitry for interactingwith the antenna(s) to cause transmission of signals via the antenna(s)or to handle receipt of signals received via the antenna(s).

Migration module 210 includes hardware configured to provide aconfigurable and incremental database migration framework forheterogeneous databases and perform/facilitate database (e.g.,heterogeneous database) migration. The migration module 210 may utilizeprocessing circuitry, such as the processor 202, to perform theseactions. The migration module 210 may receive the data via a networkinterface provided by the communications circuitry 208. However, itshould also be appreciated that, in some embodiments, the migrationmodule 210 may include a separate processor, specially configured fieldprogrammable gate array (FPGA), or application specific interfacecircuit (ASIC) to perform the database migration and/or provide databasemigration framework. The migration module 210 is therefore implementedusing hardware components of the apparatus configured by either hardwareor software for implementing these planned functions.

As will be appreciated, any such computer program instructions and/orother type of code may be loaded onto a computer, processor or otherprogrammable apparatus's circuitry to produce a machine, such that thecomputer, processor other programmable circuitry that execute the codeon the machine create the means for implementing various functions,including those described herein.

It is also noted that all or some of the information presented by theexample displays discussed herein can be based on data that is received,generated and/or maintained by one or more components of apparatus 200.In some embodiments, one or more external systems (such as a remotecloud computing and/or data storage system) may also be leveraged toprovide at least some of the functionality discussed herein.

As described above and as will be appreciated based on this disclosure,embodiments of the present invention may be configured as methods,mobile devices, backend network devices, and the like. Accordingly,embodiments may comprise various means including entirely of hardware orany combination of software and hardware. Furthermore, embodiments maytake the form of a computer program product on at least onenon-transitory computer-readable storage medium having computer-readableprogram instructions (e.g., computer software) embodied in the storagemedium. Any suitable computer-readable storage medium may be utilizedincluding non-transitory hard disks, CD-ROMs, flash memory, opticalstorage devices, or magnetic storage devices.

Exemplary Operations for Implementing Embodiments of the PresentInvention

In some embodiments, the system may be configured to a configurable andincremental database migration framework to facilitate heterogeneousdatabase migration. FIGS. 3 and 4 illustrate exemplary processes forreceiving, for example, user input identifying at least one sourcedatabase and one destination database, and, upon reception of a command,performing heterogeneous database migration. FIG. 3 shows an examplemethod that may be executed by one or more machines, for example byapparatus 200, including migration module 210 of FIG. 2, forfacilitating database migration, in accordance with some embodimentsdiscussed herein.

Providing a Configurable and Incremental Database Migration Framework

FIG. 3 illustrates a flow diagram depicting an example of a process 300for facilitating database migration in accordance with embodiments ofthe present invention. The process illustrates how, upon reception ofinformation related to source database(s) and destination (or targetdatabase(s)), a query may be generated and database migration may beperformed. The process 300 may be performed by an apparatus, such as theapparatus 200 described above with respect to FIG. 2.

As shown in block 310 of FIG. 3, an apparatus, such as apparatus 200,may be configured for checking or otherwise confirming connectivity oraccessibility to one or more databases, including in some embodiments, asource database 320 and a destination database 330. For example, theapparatus may be configured to confirm connectivity via any networkconnection (e.g., WIFI, hardwired or cellular). In some embodiments, toconfirm connectivity/accessibility, as shown in block 340 of FIG. 3, theapparatus, such as apparatus 200, may be configured for accessing,receiving, or otherwise determining connection related information foreach of a plurality of databases, including, for example, a firstdatabase having a first database type (e.g., the first database being asource database and the first database type being the type of database)and a second database having a second database type (e.g., the seconddatabase being a destination database and the second database type beingthe type of the second database). In some embodiments, block 310 mayinvolve checking three or more connections and block 340 may includeadditional database types. For example, the apparatus may supportheterogeneous database migration from a source database to a pluralityof destination database or, in some embodiments, from a plurality ofsource databases to one or more destination databases. While thedescription refers to a source database and source database type, one ofordinary skill in the art would appreciate that the present inventionmay facilitate database migration from, for example, CSV, XML, or JSONfiles as well as any standard that facilitates data interchange.

Once the apparatus is provided with connection information to eachdatabase and has confirmed connectivity/accessibility, the process 300proceeds to block 350, where a query may be generated. That is, as shownin block 350 of FIG. 3, an apparatus, such as 200, may be configured forframing a query. The query may be framed based on types of databasesinvolved in the data migration. For example, the query may be a functionof the first database type and the second database type. Where more thanone source database and/or more than one destination databases areinvolved, in some embodiments, a plurality of queries may be framed.That is, each query may be directed to a particular source database, aparticular destination database, or in some embodiments, to a particularcombination of a particular source database and a particular destinationdatabase.

In some embodiments, framing the query may involve determiningconfiguration data to generate a mapping. As shown in block 355 of FIG.3, an apparatus, such as 200, may be configured for determiningconfiguration data to generate a mapping. That is, a process of datamigration may comprise analyzing the source database and the targetdatabase to produce configuration data, the configuration data beingused for generating a mapping of the source database to a targetdatabase, which may enable the migration of, for example, at least onetable, or portion thereof, from the source database to the targetdatabase.

Block 360 shows the mapping process. For example, as shown, for each ofa plurality of tables in the target database (i.e. a target table), thesource tables/columns are identified. Specifically, as shown, for eachof N target tables, the sources table and columns are identified in themapping process. Once the query is framed, the query is executed. Asshown in block 370 of FIG. 3, an apparatus, such as 200, may beconfigured for executing query and/or fetching data, from, for example,source database 320. Subsequently, as shown in block 375 of FIG. 3, anapparatus, such as 200, may be configured for using user definedfunction(s) to process data. For example, as shown in more detail belowand in particular in FIG. 5, the apparatus may be configured forsupporting various user defined functions such as supporting constantvalues, manipulating values of specified fields, identifying multiplesource databases, parallel execution, and maintaining referentialintegrity. That is, as shown in block 380 of FIG. 3, an apparatus, suchas 200, may be configured for determining function(s) to process in datain required format and as shown in block 385 of FIG. 3, an apparatus,such as 200, may be configured for determining functions ranging fromconstant to formulae.

After querying the source database(s) and processing the data utilizingthe user defined functions, the data may be, for example, converted intoa format consistent with insertion into the destination database(s).That is, as shown in block 390 of FIG. 3, an apparatus, such as 200, maybe configured for framing data into, for example, SQL insert format.Finally, as shown in block 395 of FIG. 3, an apparatus, such as 200, maybe configured for storing the data in target database 330. In someembodiments, the insert format is achieved by identifying the sourcedatabase type and converting the data to that of destination databasetype by, for example, .calling one of a plurality of functions, eachfunction configured for converting data of one type to another. Forexample, a first function that is called when JSON data is beingmigrated to MySQL, a second for when MySQL is being migrated toPostgreSQL, etc. For example, in an embodiment in which source data isof a first type (e.g., JSON), the configuration file may comprise a flagor the like stating “is the input file of the first file type”, and ifyes or in an instance in which a yes is returned, while running themigration program, a function may be called which, for example, takesthe file name as an input and within the function, the data of the firstfile type is transferred or written directly to the target database. Thefunction name will be similar to processSQLFile similar likeprocessJSONFile public function processJSONFile($SQLFileName){

. . .

. . .

. . .

}

Database Migration

FIG. 4 illustrates a flow diagram depicting an example of a process 400for facilitating database migration in accordance with embodiments ofthe present invention.

Specifically, process 400 illustrates an embodiment in which aconfigurable and incremental database migration framework may be used toperform heterogeneous database migration. The process 400 may beperformed by an apparatus, such as the apparatus 200 described abovewith respect to FIG. 2 and database connection/data migration may takeplace via any network connection (e.g., WIFI, hardwired or cellular).

As will be shown below, a migration configuration file may be utilizedto identify source database(s) and destination databases, the migrationconfiguration file comprises information identifying one or more sourcedatabases and one or more destination databases. To generate themigration configuration file, database connection information may befirst received. As shown in block 410 of FIG. 4, an apparatus, such asapparatus 200, may be configured for receiving source databaseconnection information. As shown in block 420 of FIG. 4, an apparatus,such as apparatus 200, may be configured for receiving destinationdatabase connection information. In some embodiments, in an instance inwhich data may be migrated to (and/or from) a plurality of databases,one or more destination database connections (and/or one or more sourcedatabase connections) may be specified. In an exemplary embodiment, toreceive the source and/or destination database connection information,the apparatus may be configured to provide, for display at a userdevice, a user interface comprised of user-fillable and/oruser-selectable portions. For example, the user interface may beconfigured to display user-fillable and/or user-selectable portionsconfigured to receive information regarding a username and password. Theapparatus may then be configured to receive and subsequently store theuser input. For example, a user may input the username and password,which the apparatus subsequently stores in a migration configurationfile. In some embodiments, as shown in FIG. 3 above, the apparatus maybe configured to check the database configuration information to, forexample, confirm accessibility.

As shown in block 430 of FIG. 4, an apparatus, such as apparatus 200,may be configured for accessing, for example, the migrationconfiguration file, to read and/or identify connection information forthe one or more source database(s) and one or more destinationdatabase(s). As shown in block 440 of FIG. 4, an apparatus, such asapparatus 200, may be configured for reading migration configurationfile (e.g., migration_configuration.php). In some embodiments, themigration configuration file generated may be generated locally, whereasin other embodiments, the migration configuration file may be generatedat a remote server. Similarly, the migration configuration file may bestored locally, whereas in other embodiments, the migrationconfiguration file may be stored remotely. That is, the migrationconfiguration file may be generated and/or stored at the device at whichthe user is directing the process or at a remote server that, forexample, hosts the heterogeneous database migration framework.

In some embodiments, a connection file may be different than themigration configuration file and may comprise connection relatedinformation (username, passwords, etc.) In an exemplary embodiment,security requirements may require generation and/or storage of one orboth either locally or remotely. During execution, a call to one or bothof the connection file or the migration configuration file may be made.

In some embodiments, the types of databases may be different. Forexample, a first database (i.e. the type of the first database) may be,for example, My SQL while the second database type (i.e. the type of thesecond database) may be, for example, PostgreSQL. While embodimentsdisclosed herein discuss various methods that may be employed in theevent that the databases are of different types, it should be understoodthat the databases may also, in some embodiments, be of the same orsimilar type.

In an exemplary embodiment, a function in which the source databaseconnection is defined as get_ID_connection( )and the function in whichthe destination database connection(s) is defined as get_CD_connection() Specifically, the functions may be defined as

<?php function get_ID_connection( ) {   $tw_mysql_db_hostname =“localhost”;   $tw_mysql_db_username = “root”;   $tw_mysql_db_password =“R00T321”;   $tw_mysql_db_database = “webgroupon”;   // Check TW DBconnection   $my_con = new mysqli($tw_mysql_db_hostname,  $tw_mysql_db_username, $tw_mysql_db_password);  $my_con→set_charset(‘utf8’);   if ($my_con) {    return $my_con;   }else {     echo “Connection failed with Indonesia     database.”.$my_con→connect_error;   }  }  functionget_CD_connection( ) {   //CD DB Config setting  $cd_postgres_db_hostname = “localhost”;   $cd_postgres_db_username =“stardeals”;   $cd_postgres_db_password = “stardeals”;  $cd_postgres_db_name=“t_stardeals_asia”;   // Check CD DB connection  $conn_string = “host=cd_postgres_db_hostgame port=5432      dbname=$cd_postgres_db_name       user=$cd_postgres_db_username  $pg_con = pg_connect($conn_string);   if ($pg_con) {    return$pg_con;   } else {     echo “Connection failed with CityDealsdatabase.”;   } }

As shown in block 450 of FIG. 4, an apparatus, such as apparatus 200,may be configured for mapping tables and/or traversing each table andcolumn level configuration. That is, details regarding source databaseconfiguration and/or destination database configuration may identifiedand implemented. As such, depending on one or more particularrequirements, the apparatus may be configured to perform the sourcedatabase configuration and/or destination database configuration. FIG. 5shows a flow diagram depicts an example process 500 identifying a numberof uses cases for which configuration may be performed, in accordancewith embodiments of the present invention.

The process 500 may be performed by an apparatus, such as the apparatus200 described above with respect to FIG. 2. As shown in block 510 ofFIG. 5, an apparatus, such as apparatus 200, may be configured formigrating data from, for example, SQL, CSV, or JSON. For example, theapparatus may be configured to enable specification and/or specify adifferent input format to import data to the destination from a file.

-   -   ‘std_data.appdomains’⇒array (        -   ‘migrateData’⇒true,        -   ‘isSQLFile’⇒true,        -   ‘SQLFileName’⇒“std_data__appdomains.sql”    -   ),

As shown in block 520 of FIG. 5, an apparatus, such as apparatus 200,may be configured for supporting a constant value. For example, as shownbelow, the apparatus may be configured to specify a constant value forsome of the fields in configuration.

-   -   ‘u_appdomain_id’⇒array(        -   ‘mapping’⇒‘40 ’,        -   ‘isConstant’⇒true,    -   ),

As shown in block 530 of FIG. 5, an apparatus, such as apparatus 200,may be configured for supporting manipulation of values before migratingto the destination table. That is, for example, as shown below, theapparatus may be configured to apply one or more different functions andmanipulate a value for each column for each row before inserting it intodestination table.

-   -   ‘dm_merchant_welcome_message’⇒array(        -   ‘mapping’⇒‘webgroupon.companyinfo.companyinfo’,        -   ‘function’⇒‘stripTo250’,    -   ),

As shown in block 540 of FIG. 5, an apparatus, such as apparatus 200,may be configured for supporting data from multiple databases. Forexample, as shown below, the apparatus may be configured to join tables,take data from different source database(s) and take selected fields tomap to specified fields in the destination table.

-   -   ‘from’⇒‘FROM webgroupon.promo’,    -   ‘join’⇒‘LEFT JOIN webgroupon.ptc_city_mapping ON webgroupon.pr.        ‘LEFT JOIN webgroupon.auto_inc_mapping_deal_merchant    -   ‘where’⇒‘WHERE 1 AND webgroupon.promo.id_company=1’,    -   ‘order_by’⇒‘ORDER BY webgroupon.prome.id_promo DESC’,

As shown in block 550 of FIG. 5, an apparatus, such as apparatus 200,may be configured for supporting parallel execution. For example, asshown below, the apparatus may be utilized to execute database migrationin parallel in batch by, for example, applying a limit (e.g., to thenumber of parallel migrations), which may increase the speed ofprocessing.

‘from’⇒‘FROM webgroupon.purchase’,

‘join’⇒‘LEFT JOIN webgroupon.auto_inc_mapping_users ON webgroupon,” LEFTJOIN webgroupon.auto_inc_mapping_city_deals ON webgroupon,” LEFT JOINwebgroupon.paymenttrack ON webgroupon.paymenttrack,” LEFT JOINwebgroupon.auto_inc_mapping__users_billing_history ON webgroupon‘where’⇒‘WHERE 1’,//‘order+by’+>‘ORDER BY webgroupon.datauser.id DESC’,‘limit’⇒‘LIMIT 0,5’,

As shown in block 560 of FIG. 5, an apparatus, such as apparatus 200,may be configured for maintaining referential integrity for a primaryand foreign key in the destination database. For example, as shownbelow, in some embodiments, the apparatus may be configured to create atemporary table to maintain a relation between a primary key in a sourcetable and a destination table, which may be used to maintain referentialintegrity in destination table during migration.

-   -   ‘from’⇒‘FROM webgroupon.datauser’,    -   //‘join’⇒‘LEFT JOIN webgroupon.user_detaildata ON        webgroupon.dataus    -   ‘where’⇒‘WHERE 1’,    -   ‘order_by’⇒‘ORDER BY webgroupon.datauser.id_user DESC’,    -   ‘limit’⇒‘LIMIT 0,5’,    -   ‘auto_inc_mapping_table’⇒‘webgroupon.auto_inc_mapping_users’

In some embodiments, according to a mapping table, the apparatus may beconfigured to generate a query which may be configured to refer aprimary key from a auto increment table (auto_inc_mapping_users).

A utility library (or function library) may be provided/utilized todefine functions for utilization in migration configuration mappingwhich may be used to manipulate field values or in some embodiments,manipulate data values stored in a source database table or any dataexchange formats (e.g., XML, JSON, CSV, etc.) in advance of causingstorage of the data in the destination database table. The complexity ofthe manipulation can range from simple to more complex. For example, asshow below, the function library may include a function to modify abirth date to a required format. Based on the source database type andthe destination database type, the apparatus may be configured todetermine whether to call this particular function. That is, where datamigration is being performed and the apparatus identifies thatparticular information (e.g., a birth date) is stored in a firstparticular format in the source database, a function may be called inthe mapping process to manipulate the field values so to be able to bestored in a second particular format as required by the destinationdatabase.

As also shown below, the function library may include a function toconvert Taiwanese characters (e.g., a gender as indicated in TW) to agender in English (e.g., M (male) or F (female)). Furthermore, thelibrary may include a function, also shown below, that may be utilizedto reduce the amount of characters to a given number (e.g., 250). As oneof ordinary skill in the art would appreciate, the utility library mayinclude any function enabling the manipulation of data from a sourcedatabase to an insert format required by the destination database.

//Function to get the birthdate in required format functionsetUserBirthday(&$row) {  if(!empty($row[‘skip_u_birthday_year’]) &&   !empty($row[‘skip_u_birthday_month’]) &&   !empty($row[‘skip_u_birthday_day’])   $row[‘u_birthday’] =$row[skip_u_birthday_year’].’-     ‘.$row[skip_u_birthday_month’].’-    ‘.$row[skip_u_birthday_day’].   }   $row[‘u_birthday’] =‘2014-04-23’;   //$row[‘u_created’] = ‘2015-04-22 10:43:23.518456+00’; }  //Function to convert TW gender to CD m / f value  functionsetDefaultSex(&$row) {   $row[‘u_sex’] = empty($row[‘u_sex’]) ? ‘m’ :$row[‘u_sex’];   if(strlen($row[‘u_sex’]) != mb_strlen($row[‘u_sex’],‘utf-8’)) {   }  }  function stripTo250(&$row){  $row[‘dm_merchant_welcome_message’] =    substring($row[‘dm_merchant_welcome_message’], 0,     250);  }

Returning back to FIG. 4, as shown in block 460 of FIG. 4, an apparatus,such as apparatus 200, may be configured for creating/generating a queryon each of one or more source database(s). As shown in block 470 of FIG.4, an apparatus, such as apparatus 200, may be configured for executingthe query on the source database(s). As shown in block 480 of FIG. 4, anapparatus, such as apparatus 200, may be configured for populating oneor more temporary table(s) if, for example, table is having primary keyto maintain referential integration. An example query is shown below andrefers to a temporary table.

    SELECT qpod_full.auto_inc_mappinq_users.new_auto_inc_id ASucd_user_id, qpod_full.user_address.zip AS ucd_postal_code,qpod_full.user_address.city AS ucd_cityname,SUBSTRING(CONCAT(qpod_full.user_address.address1, ″, ″,qpod_full.user_address.address2), 1, 256) AS ucd_streetname,qpod_full.user_address.address1 AS ucd_street_number,SUBSTRING(qpod_full.user_address.tel, 1, 32) AS ucd_phone_number,qpod_full.user_address.reg_time AS ucd_last_modified, ‘false’ ASucd_mobile_verificated, qpod_full.user_address.reg_time AS ucd_createdFROM qpod_full.user_address JOIN qpod_full.auto_inc_mapping_users ONqpod_full.user_address.user_id =qpod_full.auto_inc_mappinq_users.orig_auto_inc_id WHEREqpod_full.user_address.is_master =‘YES’AND qpod_full.user_address.is_pub= ‘YES’ AND (qpod_full.user_address.reg_time >= ‘2016-01-31 00:00:00’)ORDER BY qpod_full.user_address.user_id ASC

As will be appreciated, computer program code and/or other instructionsmay be loaded onto a computer, processor or other programmableapparatus's circuitry to produce a machine, such that execution of thecode on the machine by the computer, processor, or other circuitrycreates the means for implementing various functions, including thosedescribed herein.

As described above and as will be appreciated based on this disclosure,embodiments of the present invention may be configured as methods,mobile devices, backend network devices, and the like. Accordingly,embodiments may comprise various means including entirely of hardware ora combination of software and hardware. Furthermore, embodiments maytake the form of a computer program product on at least onecomputer-readable storage medium having computer-readable programinstructions (e.g., computer software) embodied in the storage medium.Any suitable computer-readable storage medium may be utilized, includingnon-transitory hard disks, CD-ROMs, flash memory, optical storagedevices, magnetic storage devices, or the like.

Embodiments of the present invention have been described above withreference to block diagrams and flowchart illustrations of methods,apparatuses, systems and computer program products. It will beunderstood that each block of the circuit diagrams and processflowcharts, and combinations of blocks in the circuit diagrams andprocess flowcharts, respectively, can be implemented by various meansincluding computer program instructions. These computer programinstructions may be loaded onto a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the computer program product includes theinstructions which execute on the computer or other programmable dataprocessing apparatus create a means for implementing the functionsspecified in the flowchart block or blocks.

These computer program instructions may also be stored in acomputer-readable storage device that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablestorage device produce an article of manufacture includingcomputer-readable instructions for implementing the function discussedherein. The computer program instructions may also be loaded onto acomputer or other programmable data processing apparatus to cause aseries of operational steps to be performed on the computer or otherprogrammable apparatus, thereby producing a computer-implemented processsuch that the instructions executed on the computer or otherprogrammable apparatus cause performance of the steps and therebyimplement the functions discussed herein.

Accordingly, blocks of the block diagrams and flowchart illustrationssupport combinations of means for performing the specified functions,combinations of steps for performing the specified functions and programinstruction means for performing the specified functions. It will alsobe understood that each block of the circuit diagrams and processflowcharts, and combinations of blocks in the circuit diagrams andprocess flowcharts, can be implemented by special purpose hardware-basedcomputer systems that perform the specified functions or steps, orcombinations of special purpose hardware and computer instructions.

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art to which theseembodiments of the invention pertain having the benefit of the teachingspresented in the foregoing descriptions and the associated drawings.Therefore, it is to be understood that the embodiments of the inventionare not to be limited to the specific embodiments disclosed and thatmodifications and other embodiments are intended to be included withinthe scope of the appended claims. Although specific terms are employedherein, they are used in a generic and descriptive sense only and notfor purposes of limitation.

The invention claimed in:
 1. An apparatus for facilitating heterogeneousdatabase migration, the apparatus comprising at least one processor andat least one memory including computer program code, the at least onememory and the computer program code configured to, with the processor,cause the apparatus to at least: generate a utility library to definefunctions for utilization in migration configuration mapping, includingfunctions for manipulating field values and manipulating data valuesstored in each of a plurality of source database tables and any dataexchange formats in advance of causing storage of data in each of aplurality of destination database tables; generate a migrationconfiguration file by receiving connection information for accessingeach of one or more source databases, receiving connection informationfor accessing each of one or more destination databases, and confirmingconnectivity to each of the one or more source databases and each of theone or more destination databases, the migration configuration filecomprising information indicative of at least a particular sourcedatabase type of the one more source databases, connection informationof accessing the particular source database, a destination database typeof a particular destination database of the one more destinationdatabases, and connection information for accessing the particulardestination database, wherein the source database type comprises sourcedata of a first type and the destination database type specifies sourcedata of a second type different from the first type; generate each of aplurality of queries needed to facilitate the heterogeneous databasemigration, each query of the plurality of queries being a function ofthe source database type and the destination database type, and whereineach query is directed to a particular combination of the particularsource database and the particular destination database, whereingeneration of the plurality of queries comprises: analyzing the sourcedatabase and the target database to produce configuration data, theconfiguration data used for generating a mapping to enable the migrationof at least one table, or portion thereof, from the source database tothe target database; wherein generation of the mapping comprises: foreach of the plurality of destination database tables, identify one ormore particular source database tables and particular columns of each ofthe one or more particular source database tables from which the sourcedata will be migrated; based on the source database type and thedestination database type, determine whether to call a particularfunction of the plurality of functions defined in the utility library asable to, manipulate particular information stored as the source data ofthe first type in the one or more particular source database tables andparticular columns for migration to the destination database table asthe second type; maintain, during migration, referential integrity bycreating a temporary table configured to maintain a relation between aprimary key in the source database table and a foreign key in thedestination database table; and apply one or more different functionsfor supporting data migration from the plurality of source databasetables, each of the one or more different functions configured forsupporting data migration from at least one of the plurality of sourcedatabase tables.
 2. The apparatus according to claim 1, wherein the atleast one memory and the computer program code are further configuredto, with the processor, cause the apparatus to: execute each of aplurality of queries to extract data; frame the data into an insertformat; and cause storage of the data, in the insert format, in thedestination database.
 3. The apparatus according to claim 2, wherein theat least one memory and the computer program code configured for framingthe query is further configured to, with the processor, cause theapparatus to: call a function for manipulating data values stored in thesource database table or any data exchange formats in advance of causingstorage of the data in the destination database table.
 4. A method forfacilitating heterogeneous database migration, the method comprising:generating a utility library to define functions for utilization inmigration configuration mapping, including functions for manipulatingfield values and manipulating data values stored in each of a pluralityof source database tables and any data exchange formats in advance ofcausing storage of data in each of a plurality of destination databasetables; generating a migration configuration file by receivingconnection information for accessing each of one or more sourcedatabases, receiving connection information for accessing each of one ormore destination databases, and confirming connectivity to each of theone or more source databases and each of the one or more destinationdatabases, the migration configuration file comprising informationindicative of at least a particular source database type of the one moresource databases, connection information of accessing the particularsource database, a destination database type of a particular destinationdatabase of the one more destination databases, and connectioninformation for accessing the particular destination database, whereinthe source database type comprises source data of a first type and thedestination database type specifies source data of a second typedifferent from the first type; generate each of a plurality of queriesneeded to facilitate the heterogeneous database migration, each query ofthe plurality of queries being a function of the source database typeand the destination database type, and wherein each query is directed toa particular combination of the particular source database and theparticular destination database, wherein generation of the plurality ofqueries comprises: analyzing the source database and the target databaseto produce configuration data, the configuration data used forgenerating a mapping to enable the migration of at least one table, orportion thereof, from the source database to the target database;wherein generation of the mapping comprises: for each of the pluralityof destination database tables, identify one or more particular sourcedatabase tables and particular columns of each of the one or moreparticular source database tables from which the source data will bemigrated; based on the source database type and the destination databasetype, determine whether to call a particular function of the pluralityof functions defined in the utility library as able to, manipulateparticular information stored as the source data of the first type inthe one or more particular source database tables and particular columnsfor migration to the destination database table as the second type;maintaining, during migration, referential integrity by creating atemporary table configured to maintain a relation between a primary keyin the source database table and a foreign key in the destinationdatabase table; and applying one or more different function calls forsupporting data migration from the plurality of source database tables,each of the one or more different functions configured for supportingdata migration from at least one of the plurality of source databasetables.
 5. The method according to claim 4, further comprising:executing each of a plurality of queries to extract data; framing thedata into an insert format; and causing storage of the data, in theinsert format, in the destination database.
 6. The method according toclaim 5, wherein framing the query comprises: calling a function formanipulating data values stored in the source database table or any dataexchange formats in advance of causing storage of the data in thedestination database table.
 7. A computer program product forfacilitating heterogeneous database migration, computer program productcomprising at least one non-transitory computer-readable storage mediumhaving computer-executable program code instructions stored therein, thecomputer-executable program code instructions comprising program codeinstructions for: generating a utility library to define functions forutilization in migration configuration mapping, including functions formanipulating field values and manipulating data values stored in each ofa plurality of source database tables and any data exchange formats inadvance of causing storage of data in each of a plurality of destinationdatabase tables; generating a migration configuration file by receivingconnection information for accessing each of one or more sourcedatabases, receiving connection information for accessing each of one ormore destination databases, and confirming connectivity to each of theone or more source databases and each of the one or more destinationdatabases, the migration configuration file comprising informationindicative of at least a particular source database type of the one moresource databases, connection information of accessing the particularsource database, a destination database type of a particular destinationdatabase of the one more destination databases, and connectioninformation for accessing the particular destination database, whereinthe source database type comprises source data of a first type and thedestination database type specifies source data of a second typedifferent from the first type; generate each of a plurality of queriesneeded to facilitate the heterogeneous database migration, each query ofthe plurality of queries being a function of the source database typeand the destination database type, and wherein each query is directed toa particular combination of the particular source database and theparticular destination database, wherein generation of the plurality ofqueries comprises: analyzing the source database and the target databaseto produce configuration data, the configuration data used forgenerating a mapping to enable the migration of at least one table, orportion thereof, from the source database to the target database;wherein generation of the mapping comprises: for each of the pluralityof destination database tables, identify one or more particular sourcedatabase tables and particular columns of each of the one or moreparticular source database tables from which the source data will bemigrated; based on the source database type and the destination databasetype, determine whether to call a particular function of the pluralityof functions defined in the utility library as able to, manipulateparticular information stored as the source data of the first type inthe one or more particular source database tables and particular columnsfor migration to the destination database table as the second type;maintaining, during migration, referential integrity by creating atemporary table configured to maintain a relation between a primary keyin the source database table and a foreign key in the destinationdatabase table; and applying one or more different function calls forsupporting data migration from the plurality of source database tables,each of the one or more different functions configured for supportingdata migration from at least one of the plurality of source databasetables.
 8. The computer program product according to claim 7, whereinthe computer-executable program code instructions further compriseprogram code instructions for: executing each of a plurality of queriesto extract data; framing the data into an insert format; and causingstorage of the data, in the insert format, in the destination database.9. The computer program product according to claim 8, wherein thecomputer-executable program code instructions configured for framing thequery further comprise program code instructions for: calling a functionfor manipulating data values stored in the source database table or anydata exchange formats in advance of causing storage of the data in thedestination database table.